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ePortfolios and other engaged learning experiences can have extensive impact on students in many 
facets of their lives, such as subject-area learning, skill and competence development, perspectives 
on “how the world works,” and even students’ own identities, confidence, and needs. Assessing 
these various impacts can be a challenge for faculty and program developers. Existing methods can 
determine students’ attainment of competence, but very little guidance exists to help leaders 
determine the impact of their programs on the affective aspects of learning, such as changes in 
perspective and identity. This paper describes the Transformation Rubric for Engaged Learning, a 
tool and methodology for examining a program’s impact beyond competency attainment in a 
systematic, replicable manner. The Rubric can be used in addition to existing program assessment 
methods to give a more complete assessment of a variety of programs, from reflective ePortfolios to 
community service projects to degree-granting programs. 


Students often report that reflection or other 
engaged learning activities “changed my life” or “made 
me look at the world differently.” Ironically, although 
academics provide students with tools for making sense 
of life-altering experiences (e.g., reflective portfolios), 
they still lack tools to help articulate and measure the 
impact of these experiences on students. The University 
of Michigan Dental Hygiene Degree Completion E- 
Learning Program (i.e., the E-Learning Program) has 
made a preliminary step in this direction, including the 
development of a rubric and methodology for defining 
and measuring changes in student confidence, skills, 
identity, and perspective on the world. This rubric and 
method can be used by institutions to measure the 
impact of any engaged learning experience, including 
academic programs, internships, service learning 
projects, co-curricular activities, theses and capstone 
projects, portfolios, or any other learning experience 
through which open-ended student responses are 
gathered. This paper describes the tool, its use, and 
preliminary results from the E-Learning Program. 

Program Information 

The E-Learning Program was launched in 2008 to 
offer an online option to an on-campus program that 
had existed for over 30 years. Degree completion 
programs provide the opportunity for dental hygienists 
with associates’ degrees and professional licenses to 
complete the necessary remaining credit hours towards 
their bachelors’ degree. The online delivery of this 
program afforded the opportunity to develop a new 
curriculum focusing on expanding dental hygienists’ 
scholarship from basic clinical practice to include 
health promotion, community program development, 
clinical teaching experience, research, advocacy, and 
leadership. The E-Learning Program, which culminates 
in a Bachelor of Science in Dental Hygiene, comprises 


11 courses over 2 years. Didactic instruction and course 
interactions happen online, and there are three 
opportunities for students to engage with their 
communities and apply their learning: a community 
project, a teaching practicum, and a capstone project. 
There are no lectures and no tests; instead, program 
competencies are measured through a variety of 
assignment types, including papers, projects, short case 
studies, and group work. 

A key feature of the E-Learning Program is the 
integration of reflection and meta-reflection throughout 
the program, largely via an ePortfolio. It is important to 
note that the program does not measure student 
competence via ePortfolio reflections because 
competence is measured through other assignments, as 
described above. Instead, reflections ask students to 
explore their perceptions of their own abilities and their 
thoughts and feelings about their experiences. Students 
reflect on four program competencies at the end of each 
course, reflect on course goals twice per course, and 
may also complete additional reflections as determined 
by individual courses. Reflections are graded for the 
quality of a student’s writing and depth of critical self- 
assessment. 

In the final course of the E-Learning Program, 
students re-read all reflections to date and write “meta- 
reflections” on program competencies. They also 
develop final self-presentation portfolios, which include 
samples of work and synopses of students’ individual 
skills and strengths. This showcase portfolio gives 
students the opportunity to articulate their achievements 
to an audience of their choice, typically a potential 
employer or graduate school admissions committee. As 
with the in-course reflections, grades are assigned based 
on the quality of the reflective writing; the portfolio 
itself is not viewed as proof of competence. 

An extensive program assessment plan was in 
place and analyses were performed since the first year 
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of the E-Learning Program (Gwozdek, Springfield, 
Peet, & Kerschbaum, 2011). The first rounds of 
analysis showed the program to be academically 
rigorous (Springfield, Gwozdek, Peet, & Kerschbaum, 
2012). However, that analysis found that competence- 
based program assessments did not adequately capture 
students’ repeated assertions that the E-Learning 
Program changed their lives (Springfield et al., 2012). 
This paper describes the effort to investigate the 
affective, personally transformative aspects of the 
program in a more systematic, replicable fashion. 

The authors of this paper include an instructional 
designer with expertise in planning and assessing online 
education (Author 1); a consultant with expertise in 
qualitative analysis, who led the rubric development 
and served as one of three unaffiliated data coders 
(Author 2); and the program director, who also taught 
several of the courses within the program (Author 3). 

Need for a Rubric 

The central problem facing those trying to 
articulate the benefits of rich, reflective, engaged 
learning programs is that observations of students, 
student learning, and student comments tell us that 
“something special’’ is happening in our programs. This 
“something” goes far beyond the types of skills and 
knowledge one would expect students to gain in a 
typical program of study. Students say of the program 
“it changed my life” or “I see the world in an entirely 
new way now.” Even mature students—those in the E- 
Learning Program are coming back to college after an 
average of 7 years in professional practice—with 
personalities not generally given to exaggeration, report 
that “This is the best thing I’ve ever done,” and “I 
didn’t really understand at first but after the last round 
of reflections, I really started to get it why we are doing 
all these extra things.” But what is “it”? Students and 
faculty alike—in this program and others in 
academia—agree that there is an “it,” but are at a loss 
for defining it and explaining why it is important. Until 
we define and have a way of measuring “it,” we cannot 
begin to understand the program features that make it 
happen. 

Concepts such as confidence and “I think of myself 
differently” came up frequently in focus groups, so we 
looked first for an extant tool to measure program 
impact on affective traits such as confidence and self¬ 
perception. We reviewed the VALUE Rubrics from the 
Association of American Colleges and Universities 
(Finley & Rhodes, 2013), the Metacognitive Awareness 
Inventory (Schraw & Dennison, 1994), and the 
California Critical Thinking Skills Test (Facione, 
1990). While all of these provide valuable information 
about individual students’ skills, all are fairly narrowly 
focused on specific competencies. That is, these metrics 


Transformation Rubric for Engaged Learning 64 


assess the extent to which students have attained or 
improved a skill, but they do not examine students’ 
perception of that skill or their personal reaction to 
having acquired that skill. Although skill achievement 
is certainly important, we believe that some of the most 
profound impacts of the E-Learning Program—which 
included extensive ePortfolio reflections, community 
engagement, and other authentic learning experiences— 
were not being measured by these standard assessments 
of competence. 

Looking more broadly, the literature around 
transformational learning did speak to issues of 
education’s impact on learners that goes beyond 
attainment of competence. A thorough overview of 
transformational learning was written by Baumgartner 
(2001), who defined transformational learning as 
learning that “changes the way people see themselves 
and their world” (p. 16). This gets at the heart of what 
we are trying to measure and corresponds to the 
categories of identity and perception eventually used in 
the rubric. Bandura and Schunk (1981) observed that 
skill acquisition often led to enhanced self-efficacy, 
confidence in one’s ability to succeed on a task, and the 
belief that self-efficacy has a direct impact on 
performance in terms of task perseverance. This ties 
into the idea that confidence is important because it 
impacts professional performance. 

Extensive discussion exists describing the 
importance of reflection and engagement for 
transformational learning (Mezirow, 1991; Taylor, 
2007). Both King (2004) and Taylor (2007) stated that 
reflection involving written accounts of these 
intellectual and emotional connections to learning is 
significantly important in the transformative process, as 
it strengthens a person’s analytical capabilities. Taylor 
(2007) described how higher education can foster 
transformational learning by creating active learning 
experiences that are directly related to content taught, 
are personally engaging, and stimulate reflection. 
Lastly, Kiely (2005) provided a thorough analysis of 
the educational psychology of learning, which is critical 
for understanding how to structure experiences to 
maximize student transformation. He also explained 
why reflection is a critical component of this process; 
however, it alone is not adequate to foster 
transformative learning. The literature suggests that it is 
a combination of both reflection and engagement that 
best supports transformative learning. 

All of these sources are vital in describing how 
transformative learning can happen within individuals 
and programs, and can be used prospectively during 
program planning to develop experiences with a high 
potential to foster transformative learning. However, 
literature on transformative learning does not provide 
tools for measuring those impacts directly. The ability 
to quantify the personal, affective impact of programs is 
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key to illustrating the unique effectiveness of reflective 
and engaged learning experiences. Therefore, we chose 
to develop a method for measuring these additional 
impacts. 

Purpose and Benefits of the Transformation Rubric 

In a recent editorial, Rhodes, Chen, Watson, and 
Garrison (2014) called for more rigorous research into 
the impact and effectiveness of ePortfolio. We believe 
our approach, which focuses on portfolio-supported 
engaged learning, may be one answer to that call. The 
Transformation Rubric we present is a coding 
mechanism for analyzing the impact of ePortfolios and 
other engaged learning experiences on students’ 
perceptions and understanding of their abilities, their 
motivations, and how they understand the world around 
them. This is a much broader goal than assessing “what 
students learned’’ or “are students competent?”; instead, 
it asks “how did students’ understanding of themselves 
and their abilities change?” It is concerning that many 
deeply enriching student experiences (e.g., ePortfolios) 
may be in danger of being cut if meaningful impact 
cannot be demonstrated. Further, using only rubrics that 
assess competency achievement could undermine 
efforts to demonstrate impact because the impact of 
engaged, reflective initiatives like portfolios goes far 
beyond typical notions of student learning. All learning 
brings about change; what is needed is a way to 
differentiate between transformative and non- 
transformative changes. We believe that this rubric can 
be used to determine objectively the number of changes 
in confidence, skills, worldview, personal identity, and 
pride that students report as a result of their 
experiences. Even more significantly, it could be used 
to determine how profound those changes were— 
distinguishing between expected incremental changes 
and truly transformative changes. This helps paint a 
much richer picture of the impact of portfolios and 
engaged learning experiences, one that more accurately 
portrays their ongoing impact on students’ lives. 

This rubric is being offered for the use of the 
engaged learning community. Although we hope that it 
will be used as-is, the rubric is designed so that other 
program-specific topics of interest may be added. It can 
be used to analyze both data gathered for this purpose 
and previously-collected data, allowing for 
retrospective and longitudinal analysis. The potential 
benefits to the community include the following: 

• Facilitating more objective evaluation of the 
transformational power of engaged and 
reflective learning beyond competency 
attainment; 

• Developing a common vocabulary around 
transformative change; 
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• Achieving the ability to directly compare 
effectiveness of program designs and teaching 
techniques to maximize outcomes and to 
optimize inputs of faculty time, software 
support, and etc.; 

• Increasing the power and validity of studies by 
facilitating inter-programmatic assessment; 

• Reducing dramatically the amount of time 
needed to develop assessment measures. 

Method of Development 

Focus groups were held with graduating students as 
a standard component of the E-Learning Program 
evaluation. When reviewing themes in students’ 
responses, we noticed that students spontaneously and 
repeatedly mentioned how the E-Learning Program had 
increased their confidence (e.g., Springfield et al., 
2012). Faculty discussions suggested that the majority 
of students mentioning confidence were, academically, 
middle-performing students. This spurred interest in 
developing a more formal method of analyzing the E- 
Learning Program’s impact on confidence and other 
factors, specifically comparing impacts between 
performance groups (high-, middle-, and low- 
performing students). 

The author with expertise in data analytics 
identified qualitative analysis as an appropriate 
approach to our data. Qualitative analysis is ideal for 
identifying the themes of participants’ open-ended 
responses, especially when researchers are un- or 
minimally-able to establish a set of analytic categories a 
priori from the existing literature (Auerbach & 
Silverstein, 2003; Brown, Tappan, Gilligan, Miller, & 
Angyris, 1994; Hill, Thompson, & Williams, 1997; 
Strauss & Corbin, 1994). In essence, this is a bottom-up 
approach to data analysis. Qualitative methods provide 
a systematic, documented, and audit-able structure for 
data analysis (Auerbach & Silverstein, 2003). 

This author trained the other authors in qualitative 
methods and guided the rubric development process 
described below. Our qualitative analysis relied 
primarily on Grounded Theory (Strauss & Corbin, 
1994) and borrowed from other approaches as well 
(Auerbach & Silverstein, 2003; Brown et al., 1994; Hill 
et al., 1997). 

The steps are summarized here and described in 
more detail later. The first five steps were devoted to 
identifying and creating the coding categories (i.e., 
themes), using a combination of focus group transcripts 
and similar non-target data from course assignments. 
This allowed the primary investigators to identify a set 
of categories for use by a separate team of coders. The 
coders were trained using the non-target data (Step 6), 
and then analyzed the focus group data (Steps 7-9). The 
steps include the following: 
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1. Two investigators drafted an initial list of 
likely coding categories and definitions. The 
list was derived from a combination of 
theoretical (a priori ) assumptions and items 
that emerged from the data. In essence, step 
one was to determine what “it” is that makes 
this program so impactful. This step distilled 
the essential impact down to transformative 
changes in perceptions, skills, identity, 
confidence, and pride. 

2. The first and third authors read target data and 
refined the list of categories and definitions. 
This refined list became the draft coding rubric. 

3. Using similar, but non-target, data, the second 
author used the draft coding rubric to 
categorize data to test it for comprehensibility. 

4. All three authors met to discuss and refine the 
rubric. 

5. Steps 2 through 4 were repeated as necessary 
until the codes and definitions were complete and 
understandable to people unfamiliar with the 
data. 

6. The first author was identified as the leader of 
the coding process. The coding leader coded 
several pages of text from non-target data to 
create a master key for training purposes. She 
then trained two coders unfamiliar with the 
program and its data to use the coding rubric. 
Additional clarifications, examples, and minor 
changes were made to the coding rubric during 
the training process. 

7. When each coder was able to match the key 
with 70% accuracy, the two trained coders and 
the second author (who also matched the key 
at least 70%) began coding target data. Each 
coder worked separately and submitted coded 
materials to the coding leader. 

8. Group meetings of the coding team (i.e., a 
team including the leader and all coders) were 
used to resolve any discrepancies in coding 
and arrive at a final code. Discrepancies were 
resolved by 100% consensus of all coders. To 
minimize confusion across transcripts, the 
team coded and discussed one transcript in full 
before proceeding to the next transcript. 

9. The coding leader maintained a record of all 
initial and final codes. During these meetings, 
the coding leader’s role was to maintain 
records and ensure the fidelity of the coding 
scheme. 

In practice, the coding rubric started with one item 
(confidence), quickly blossomed to over a dozen terms, 
and then was pared down to two major categories (type 
of change and degree of change), each of which had at 
least two subcategories or levels. 
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Steps 1 through 5 of process outlined above took 
over 14 months. Training the coding team required 
approximately two months and coding five focus group 
documents took ten weeks. Other institutions wishing to 
perform similar qualitative analyses can use this 
established coding rubric to begin at Step 6 (training 
coders), and thus complete the assessment in a 
relatively short period of time. 

The Transformation Rubric: Definitions and 
Examples 

Each data point (e.g., student comment in a focus 
group) was coded in two ways: (a) determining the 
degree of change and (b) defining the type of change. 
For example, a phrase may be coded as representing a 
transformative change in perception or a non- 
transformative change in skill. Type of change 
statements may be double-coded; for example, the same 
statement may indicate changes in skill and confidence. 
However, each statement can only have one degree of 
change; it is either transformative or non- 
transformative. The Appendix has a printable summary 
of this coding rubric. 

Degree of Change 

Some changes are more profound than others. In 
this rubric, we use the terms “non-transformative” and 
“transformative” to describe the difference. 

Non-transformative change. Non-transformative 
changes are changes in extent/amount; they represent a 
quantitative change from an existing amount to a 
greater amount. Keywords suggesting a non- 
transformative change include: “improved,” 

“enhanced,” “better,” “somewhat,” “to some degree,” 
and other phrases suggesting a moderate growth of an 
existing quality. For example, “I’m somewhat more 
confident than I was before” is a non-transformative 
statement because the change is moderate. 

Transformative change. Transformative changes 
are changes that have substantially altered the student in 
some way; they represent a qualitative change to an 
entirely new state. Key phrases suggesting 
transformative change include: “I now feel,” “life- 
changing,” “all the time now,” “much more/less,” “I 
used to . . . but now I . . . .” For example, “I used to 
think of myself as a ‘teeth cleaner,’ but now I know I’m 
an important part of a patient’s healthcare team” is a 
transformative statement. 

Type of Change 

Students may experience transformative and non- 
transformative changes in a number of areas. This 
rubric defines five types of changes. 
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Confidence. Confidence is the student’s perception 
of comfort or perceived ability to do something. It does 
not necessarily denote an actual improvement in skill, 
but rather greater belief in one’s ability to use a skill. 
“I’m much more comfortable treating patients with 
disabilities now” is an expression of confidence. 
Confidence is often double-coded with another type of 
change, such as skill. 

Pride. Pride describes joy in an accomplishment, 
feeling good about one’s self, satisfaction, and so on. It 
can sometimes be hard to differentiate from confidence. 
For example, “I was really proud of how much my 
writing improved” is an expression of pride. 

Skills. Skills are anything the student has learned to 
do as part of the program. This includes psychomotor 
skills (e.g., teaching instrumentation), academic skills 
(e.g., professional writing), and interpersonal skills (e.g., 
intercultural communication). For example, “I am now 
able to communicate with patients in non-technical 
terms” is an example of a student recognizing a change 
in skill. 

Perspective. Changes in perspective represent 
changes in how the student understands or sees other 
people or situations. For example, “I used to think most 
people on Medicaid didn’t care about their oral health, 
but now I understand that their poor oral health is an 
access to care issue” indicates a change in perspective. 

Identity. Changes in identity represent changes in 
how students understand or see themselves; for 
example, “I never thought of myself as a leader before, 
but now I do.” A special case of the non-transformative 
identity code is when students describe how an 
experience confirmed an existing identification, desire, 
or motivation. We interpreted these statements as 
removing at least a little doubt and thus coded them as 
non-transformative. For example, “My teaching 
practicum confirmed that I really do want to be a 
teacher” is a clear statement of identity that indicates 
some change (from an implicit uncertainty to an explicit 
certainty) but not a transformation. 

Do Not Code 

The following items are not coded. In practice, we 
found it helpful to mark passages “do not code” or 
“dnc” to eliminate confusion. Types of items that are 
not coded include the following. 

Statements about others. This rubric is interested 
with students’ recognition of their own changes; 
therefore statements about others are not coded. For 
example, one student saying to another “Your writing 
has really improved!” would not be coded. 

Statements of fact. Do not code lists of tasks 
performed unless accompanied by a clear, explicit 
statement that these were new or improved skills. For 
example, “I write well” or “In my practicum, I led 
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discussion groups and graded papers” speak only about 
the student’s state at that moment and do not explicitly 
indicate that a change has occurred. 

Generic, vague, or unspecitied change. These are 
items that indicate some level of change, but the change 
is not specific enough to fit into a category or is too 
vague to be a meaningful indicator. For example, “I’ve 
strengthened some traits and grown a lot.” 

Statements of projected future growth. Do not 
code statements such as “I will keep working on my 
writing in the future”; these are projections, not 
statements of change that have already occurred. 

Using the Transformation Rubric 

Once a rubric is decided upon (this one, a 
modification of this one, or a different rubric), the steps 
are straightforward: 

1. Select data to be analyzed. 

2. Select and train coders. 

3. Code the data. 

4. Analyze the data. 

Types of Data that Might be Analyzed 

Any open-ended, free-response data can be 
analyzed with this coding rubric. It is important that 
questions ask students to think about their growth and 
change, the value of the program, and so forth. For the 
E-Learning Program analysis, responses to focus group 
questions were used (see Table 1). This rubric could 
also be effective with: 

• Portfolio reflections 

• Student reflective essays 

• Exit interview transcripts 

• Open-ended written survey questions 

• Online threaded discussion questions 

To preserve anonymity, all identifying information 
within the data is replaced with randomly assigned 
identification numbers. A key that allows researchers to 
match coded transcript data to anonymized student 
profile data is maintained by one of the authors. 

Selecting and Training Coders 

Results will be most objective if people 
unaffiliated with the program under review are trained 
to code the data. Two or three coders are typically 
suggested (Auerbach & Silverstein, 2003; Brown et 
ah, 1994; Hill et ah, 1997; Strauss & Corbin, 1994). 
Use of people familiar with the students is possible, 
but increases the likelihood that anonymity will not be 
maintained as well as the potential for bias. 
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Table 1 

Focus Group Questions Used in the E-Learning Program 

The focus group data used to develop the coding rubric had the following questions: _ 

• Did you have any a-ha moments? 

• What does it mean to be a leader in dental hygiene? How do you think your views have changed compared to 
when you first started the program? 

• How has your understanding of either your own career opportunities or careers for hygienists in general 
changed? 

• How has your understanding changed of what it means to be a member of a multidisciplinary heath care team? 

• What do you see as the role of reflection in your profession moving ahead? 

• Do you notice differences between yourself and the people you work with vis a vis reflection? 

• Can you identify something you do differently as a result of being reflective? 

• Did your growth surprise you? _ 


To train coders, the coding leader distributes and 
explains the codes along with some practice items. 
Coders then practice individually. The coding leader 
then meets with the coders and discusses the results. 
The coding leader may need to correct coders’ 
understanding of the rubric or may need to refine the 
rubric further (or add more examples) in order to 
clarify. Repeat this process until at least 70% inter-rater 
reliability is achieved (Hill et al., 1997; Strauss & 
Corbin, 1994). Our experience suggests this will likely 
take four to six rounds of coding. During training, it is 
better to code smaller amounts of data and discuss the 
coding more frequently than to code larger samples 
with less discussion, because accuracy and inter-rater 
reliability are built through discussion and consensus. 

Preferably, coders should be trained on practice 
data, not the data to be included in the final analysis 
(Brown et ah, 1994; Hill et ah, 1997; Strauss & Corbin, 
1994). Consider using practice data in a pilot cohort 
that is not part of the full study or select a question that 
will not be included. If practice data cannot be found, 
train on a small subset of the final data, but re-code it 
after training is complete and all other data have been 
coded. For training, we used students’ written 
reflections from an online class discussion; the selected 
text addressed types of change similar to those 
mentioned in the focus groups that were the final data 
set. 

Coding 

Once inter-rater reliability has achieved a 
minimum of 70% agreement, coding of actual data can 
begin. Each coder should work alone to code the data or 
a portion of the data. Then, as in the practice rounds, 
the coders meet with the coding leader and come to 
consensus on the final code (Auerbach & Silverstein, 
2003; Hill et al., 1997). The coding leader generally 
does not interfere with the consensus process, unless 


the coders cannot reach consensus or have clearly 
misunderstood the coding rules. In this rubric, every 
coded passage must be coded for depth and type of 
change, and more than one type of change may be 
applied to each passage. For example, the same passage 
may be coded as skill, confidence, and transformative. 

At the end of the training phase, all three coders 
were in agreement for more than 70% of items. Across 
the five focus groups, the three-person agreement rate 
for our five categories fell to 33.6%; two person 
agreements accounted for another 50.6% of coded items 
(Cohen’s kappa ranged .31 to .36), indicating that we 
had agreement between at least two coders for 84.2% of 
items. On the practical level, these issues caused us to 
spend more time discussing each transcript in order to 
achieve consensus. Assessment of transformation was 
better, with a 61% rate of three-way agreement 
(Cohen’s kappa = .31-.41) across the five focus groups. 

We identified three distinct reasons for the drop in 
agreement. First, the training materials were drawn 
from an online discussion thread with more narrowly 
worded questions that rarely had passages that could be 
double-coded. In contrast, student comments in the 
focus groups lent themselves to multiple type-of-change 
codes; for example, identity and confidence. It was 
common for Coder A to mark a passage as identity, 
Coder B to mark it as confidence, and Coder C to mark 
it as both. After discussion, the group would often agree 
on a final code of identity and confidence. By strict 
count, only Coder C matched the final code 100%, even 
though the other two coders matched the final code 
50%. 

The second factor was the difficulty of deciding 
how much text to code for context. For example, Coder 
A might code all five sentences of a paragraph as skill, 
while Coders B and C only coded the last two sentences 
as skill (deeming the first three sentences to be 
contextual but not critical to code). In this case, 100% 
agreement was achieved for two sentences ,but three 



Springfield, Smiler, and Gwozdek 

sentences did not have 100% agreement, simply 
because Coder A coded more sentences for context. 

Finally, after one coder repeatedly coded passages 
as both identity and perspective, we clarified that these 
two categories were generally mutually exclusive, since 
identity focuses on the self and perspective focuses on 
other people. This resulted in increased consensus in 
the final three focus groups. Given the high rates of 
partial agreements among coders and the use of a full 
consensus procedure to reconcile disagreements, we 
believe that our results provide an accurate summary of 
students’ comments. 

Practical Issues 

Coding leaders need to decide what size unit of text 
to analyze: a phrase, a sentence, or a paragraph 
(Auerbach & Silverstein, 2003; Hill et al., 1997). In this 
analysis, we generally worked with one sentence at a 
time, unless students included multiple distinct thoughts 
in one run-on sentence. In that case, the sentence was 
divided into phrases at logical changes in topic. 

This group of coders found it easiest to track codes 
by working on individual word-processed documents, 
highlighting passages, and inserting comments with the 
code(s) in the comment (see Figure 1). This allowed the 
coding leader to know exactly where adjoining codes 
started and ended. Each coder worked on a separate 
copy of the document and then sent the coded 
document to the coding leader. 

The coding leader then transferred the codes to a 
spreadsheet containing one row for each sentence or 
phrase of the focus group transcript and columns for 
each coder’s responses. The spreadsheet can then be 
used to calculate the percentage agreement between the 
coders (see Figure 2). When coders provide multiple 
codes for the same phrase, that line should be listed 
multiple times in the spreadsheet and each code entered 
on a single line. During the consensus meeting, the 
coders decided on the final code (using their own coded 
documents for reference) and the coding leader 
recorded the final code in the spreadsheet (see Figure 
2). The spreadsheet could then calculate how often each 
coder’s original code matched the final code and gave 
some sense of the individual accuracy of each coder. 
Experience strongly suggests also noting the final code 
on paper as well as on the spreadsheet, in case of file 
corruption or loss. The coding leader should maintain 
copies of both the original coded documents and the 
final codes. 

For smaller data sets, each item should be coded by 
two to three people (Auerbach & Silverstein, 2003; Hill 
et al., 1997). For large data sets, more coders may be 
needed, but every coder does not need to review every 
document. Instead, simply assign two to three coders to 
read each document. Vary the coder groups so coders 
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A, B, and C are not always working on the same data. 
Instead, have coders A, B, and C work on one source 
and coders A, D, and G work on a second source, and 
so on. 

Analyzing the Data 

Setup. The first step toward analysis is to input all 
the data into a qualitative statistical software package 
(e.g., Nvivo). Source files (e.g., focus group transcripts) 
must be imported and marked up according to the final 
codes determined by the coders. A spreadsheet of 
student characteristics is also needed to create 
subgroups or to compute correlations, if desired. 
Typical characteristics include each student’s code 
name or number, cohort (if examining multiple groups), 
GPA, gender, ethnicity, age, and etc. Other factors of 
interest tracked included class rank grouping (1 st , 2 nd , 
and 3 rd tertile by GPA), overall portfolio reflection 
grade, years elapsed since last degree earned, and 
number of children cared for during the program. 

It may also be helpful to track data concurrently in 
a quantitative statistical package such as SPSS. This is 
helpful for computing basic quantitative measures, such 
as number of participants in pre-specified categories 
(e.g., gender or racial groups), computing correlations 
among student characteristics, or determining the best 
ways to group students. 

Analyses. Users of the Transformation Rubric may 
then perform the analyses they desire on the data. We 
assume that many users will wish to perform analyses 
such as the following: 

• Determining how frequently each type-of- 

change code appears in the data, (a) for the 

entire group and/or (b) for each subgroup (e.g., 
cohort, grade rank group, gender); 

• Determining how frequently transformative 

and non-transformative changes occur (a) for 
the entire group, (b) for each subgroup, and/or 
(c) to compare the percentage of 
transformative vs. non-transformative changes 
overall and for each type of change; 

• Determining how frequently each type of 

change is related to different parts of the 
program in order to help determine which 
parts of the program elicited the most 
(transformative) changes (a) for the entire 
group and/or (b) for each subgroup (e.g., 
cohort, grade rank group, gender). 

Because the rubric is so new, it is not yet known what 
may constitute a threshold for a transformative 
program. We hypothesize that students in all types of 
programs will report changes, but that the changes in 
highly engaged programs with reflective components 
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Figure 1 

Using Comments to Code Data 


1:1 wanted to go back to (stu 2) comment about professional writing expectations as a highlight 
of the program, pne of the things I think I grew the most in in this program was mv writing 


ability. I look back on that and I'm so glad that I had that opportunity, f appreciate the high 
expectations that this program has. |t think that ability to write at a professional level is a 
springboard for everything else were gomg to do - whether it's doing community projects or 
teaching and writing lesson plans, or whatever. I think the principle writing element is critical to 

_, - - j Comment [AI7]: Skill, TR ] 


anv future profession for students. (That was a highlight for me. 

_ , - - j Comment [AI8]: Perspective. TR 




Figure 2 

Spreadsheet for Tracking All Codes 


J Line # 

Stu# 

Text of response 

Final 


Coder 1 


Coder 2 


Coder 3 



13 

5 

But 1 really liked to be able to do the whole studen 

dne 


dne 


dne 


dne 


14 

5 

It was really cool to be able to go into a college an 

dne 


dne 


dne 


dne 


15 

5 

Usually you cant just goto a place and say "Hey, 

dne 


dne 


dne 


dne 


16 

5 

1 think the teaching was the most memorable for 

Ident 

TR 

Ident 

TR 

Ident 

TR 

Ident 

NT 

20 

4 

1 agree with 5 on that-the teaching was very eye-t 

Ident 

TR 

ident 

TR 

Ident 

TR 

Ident 

TR 

20 

4 

1 agree with 5 on that - the teaching was very eye-t 

Persp 

tR 

Ident 

TR 

Persp 

TR 

Ident 

TR 

21 

4 

The program director had me doing more logistic 

Ident 

TR 

Ident 

TR 

Ident 

TR 

Ident 

TR 

21 

4 

The program director had me doing more logistic 

Persp 

tR 

Ident 

TR 

Persp 

TR 

Ident 

TR 

22 

4 

1 had thought that 1 wanted to be a teacher prior to 

Ident 

TR 

Ident 

TR 

Ident 

TR 

Ident 

TR 

22 

4 

1 had thought that 1 wanted to be a teacher prior to 

Persp 

tR 

Ident 

TR 

Persp 

TR 

Ident 

TR 

23 

4 

I've had deal whether or not 1 want to step into a te 

Ident 

TR 

Ident 

TR 

Ident 

TR 

Ident 

TR 

23 

4 

I've had deal whether or not 1 want to step into a te 

Ident 

TR 

Ident 

TR 

Persp 

TR 

Ident 

TR 


will be relatively more transformative than standard 
lecture-test programs. 

Preliminary Results for the E-Learning Program 

Using the Transformation Rubric and 
methodologies above, the following preliminary 
results were obtained for the E-Learning Program. The 
frequency of each type of change across the first five 
cohorts of students in the E-Learning Program is 
provided. Each focus group was held around 
graduation. Out of the 1,045 statements made by the 
30 students who participated in the five focus groups, 
we identified 249 changes. Over a third of those 
changes were perspective changes (36%). Pride was 
the least-reported change (6%). On average, each 
student reported 8.3 changes. The frequency of 
transformative and non-transformative change is 
shown in Figure 3. Overall, 210 of the changes 
reported (84%) were transformative changes, with 
only 40 of the reported changes (16%) being non- 
transformative. On average, students reported seven 
transformative changes and 1.33 non-transformative 
changes. 


Conclusion and Next Steps 

Students often report that educational experiences 
have changed theirs lives but educational researchers 
have had great difficulty documenting these changes 
outside of anecdotal reports. This stands in contrast to 
efforts to document changes in students’ knowledge 
and skills, for which a variety of systems exist. In this 
paper, we have provided a tool and methodology for 
objectively measuring a program’s impacts beyond 
attainment of competence: the Transformative Rubric 
for Engaged Learning. Preliminary analysis of our E- 
Learning Program shows that over one-third of the 
changes reported by students were perspective changes, 
that is, differences in the way students perceive their 
work, patients, field, and concepts such as leadership. 
Over a quarter of the students also reported changes in 
their clinical, interpersonal, and technical skills. The 
vast majority of changes were transformative in nature, 
suggesting that graduates not only have better 
knowledge of their field but also see their field and 
themselves very differently than when they entered the 
program. 

Many ePortfolio and engaged learning programs 
stimulate dramatic changes in students’ lives, but lack a 
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Figure 3 

Type and Degree o f Changes Observed inE-Learning Program 
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way to measure that impact simply and directly. If the 
rubric becomes a widely used instrument, we expect 
three main benefits. First, programs that foster 
transformational learning will be able to measure and 
describe their students’ confidence, identities, 
perceptions, pride, and skills in addition to the 
competency assessment tools already in use. This will 
help ensure the continuance of these programs in a time 
of increasing budgetary uncertainty. Second, if 
programs’ non-competency outcomes can be measured 
in a systematic way, program inputs (e.g., faculty and 
staff time) can be optimized because the impact of 
changing inputs can be observed in program outcomes. 
Finally, with a standardized method for analysis, highly 
impactful programs can be identified and best practices 
shared, thus improving learning for students in a wide 
variety of programs. 

What is most needed now is experimentation 
within the community to use the rubric to analyze a 
variety of programs, including both programs with 
heavy emphasis on engagement and reflection and more 
traditional programs. This will allow the community to: 

• assess the validity of the Transformation 
Rubric as a tool for measuring 
transformational change across a variety of 


programs, subject areas, and student 
demographics; 

• start understanding what program features 
contribute most to actual transformative 
outcomes; and 

• start building an objective picture of the 
impact of engaged learning activities beyond 
attainment of competence. 

Ideal subjects would be programs about to undergo a 
significant increase in engaged curricula or teaching 
methodologies (e.g., introducing a reflective portfolio). 
By analyzing data from students before and after the 
change, it should be possible to see if the new 
curriculum engenders more transformative changes 
than the old program. 
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Appendix 
Printable Rubric 


This document was developed in 2013 at the University of Michigan School of Dentistry using the 
grounded theory methodology (Strauss & Corbin, 1994). It can be used by researchers to investigate the impact 
of a wide range of engaged learning opportunities, such as reflection, portfolios, service learning, clinical 
practice, and simulations. Changes in five areas are currently defined, as well as two levels of change: 
transformative and non-transformative. This rubric can be used to code data from a number of qualitative 
sources: open-ended survey questions, focus group transcripts, student essays or reflections, and so on. Once 
coded, changes can be analyzed to determine program impact. See the associated paper for details for use. 


Type-of-Change Codes 

Code and Description 

Details 

Keywords & Examples 

Confidence: Students’ 
perception of their comfort 
or perceived ability to do a 
thing 

This is NOT better skill, but rather 
greater belief in one’s ability to use a 
skill. 

Often double-coded with skill or 
identity. For example, “I am much 
more confident about my writing 
skills” would be double-coded for both 
Confidence and Skill. 

Confident, comfortable, easy, no 
longer a problem 
“I used to be terrified of public 
speaking, but now it doesn’t bother 
me.” 

“I’m much more comfortable 
networking now.” 

Pride: Expressing 
gratification in an 
accomplishment 

Similar to confidence, but usually very 
explicit about being proud of a skill. 

“I found it rewarding to...” 

“I was so proud that I...” 

Do not code generic statements such 
as “I’m so proud of all of us” 

Skills: anything the student 
has learned to DO as part 
of the program. 

Teamwork, relationships, “soft” skills 
Clinical skills 

Communication/presentation skills 
Teaching skills 

Leadership skills (I have improved my 
leadership skills, like listening, 
communicating, etc.) 

“I can now...”, 

“I have improved...”, 

“I [verb] better...” 

“I have become more [adjective]...” 

Perspective: changes in 
how the student 
understands or sees other 
people 

Externally focused 

I understand SOMETHING ELSE 
(outside of myself) differently than I 
did before 

How I perceive other people is 
different 

How I think/believe the world works is 
different 

I have a different understanding of my 
profession now 

“I see leadership in a different way; I 
used to think leadership was X, now I 
think it’s Y.” 

“I never realized how hard it is to sign 
up for Medicaid.” 

“I used to think the best way to change 
someone’s mind was to give them 
more data. Now I know it’s more 
complicated.” 

Identity: Changes in how 
students understand or see 
themselves. 

Internally focused 

Vision of self, career, path 

Traits about myself: I AM a different 
person or kind of person 

Change in motivation or direction (I’m 
now motivated to...) 

Confirmation of motivation or 
direction (non-transformative change 
only) 

“I am now...” 

“I have become a better [noun]...” 

“I see myself as a Leader; my vision of 
myself as a leader has changed” 

“I confirmed that I still want to...” 
(always non-transformative) 

Role, see myself 
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Extent-of-Change Codes 

Code and Definition 

Examples 

Keywords 

Transformative change: 

The student recognizes that 
change occurred AND the 
change has substantially 
altered them in some way. 

Qualitatively different; different in 
kind 

The way the student conceptualizes 
this issue is significantly different 
than it used to be 

“I now feel . . .” 

“I used to . . . but now I . . 

Life-changing, no longer, not anymore, 
all the time now, changed a lot, much 
more, in a different light, enlightened, 

Not transformative 
change: The student sees a 
change in extent/amount. 

Quantitatively different; different in 
amount 

No “generic” statement can be 
transformative 

Enhanced, acquired, improved, better 
than before, somewhat, to some degree, 
a little 


Do Not Code 

Items in this category will not be counted or analyzed. Use this code to clearly mark passages that should not 
be coded to avoid confusing and time-consuming re-reading. 

Statements about others 

“Your writing has really improved!” Student must be observing her own 
changes. 

“We’ve all grown” 

Statements of fact 

Lists of tasks performed, unless accompanied by a clear, explicit statement that 
these were new or improved skills. 

“I write well.” (We don’t know if she did before or not.) 

“I see myself as a leader.” (We don’t know if she did before or not.) 

“Doing X was useful.” (We are interested in whether it was also a change.) 

Any statement that highlights a fact about a current state, with no suggestion 
that a change has occurred. The coder should not infer that a change has 
happened; we are interested in whether the student herself noted the change. 

Generic, vague,or 
unspecified change 

“This program prepared me to move forward” 

“I’ve strengthened some traits and grown a lot” 

Statements of projected 
future growth 

“I’m sure we will continue to grow” 

“I will keep working on my writing in the future” 


Notes regarding context: 

You may use the context of a paragraph to determine if something is a change (vs. a statement of fact). 
You may wish to look at the paragraph as a whole and determine whether there was a substantive change or 
not; then look for specific instances line-by-line. 

Do not make assumptions based on context outside the paragraph. For example, do not make assumptions 
about students’ motives, prior abilities, or former actions. 





